1. Field of the Invention
The present invention relates to an image pickup apparatus, such as a video camera and a digital camera, and more particularly to an image pickup apparatus having a function of correcting the distortion of an image photographed by an image pickup sensor of a rolling-shutter type and an image processing method of a picked-up image.
2. Description of the Related Art
In recent years, many video cameras and digital cameras use CMOS image pickup sensors of the rolling shutter type. Each of the image pickup sensors of this type has a time lag of image pickup in each line (horizontal scanning), and consequently the image pickup sensor has the problem in which a photographed image is distorted in a diagonal direction in panning, the problem in which a photographed image is expanded or contracted in tilting, and the problem in which a photographed image is distorted into a waved form due to camera shaking.
When a still image is photographed with such a camera, the time lag of image pickup is frequently removed by sensor driving in a global-shutter method and a combination of mechanical shutters. However, these methods are not suited for photographing moving images because these methods cannot simultaneously perform exposure to a sensor and the read-out of an image from the sensor so that the interval of image pickup becomes long.
Some cameras photographing moving images by the rolling-shutter method store photographed images in buffer memories, shift a read-out position of each of the lines, clip out parts of stored images to output them, and thereby correct the distortion of photographed images. The read-out position of each of the lines is determined by detecting in advance the motion amounts of the cameras to objects with gyro sensors or the like.
For example, as illustrated by a broken line in FIG. 16A, if an object 1600 is photographed when a camera stands still, a range 1601 is photographed, and a photographed image 1602 including no distortion as illustrated in FIG. 16B is stored in the buffer memory of the camera. On the other hand, when the camera is panning to the right side, as illustrated in FIGS. 16C and 16D, a range 1611 is photographed, and a photographed image 1612 distorted in a diagonal direction is stored. Accordingly, as illustrated in FIGS. 16E and 16F, an image 1614 including corrected distortion is output by clipping out a range 1613 while shifting the read-out position of each of the lines by using a detection result of the motion amount.
Moreover, when the camera is standing still, as illustrated in FIGS. 17A and 17B, a range 1701 of an object 1700 is photographed, and a photographed image 1702 including no expansion and contraction is stored in the buffer memory. However, when the camera is tilting downwards, as illustrated in FIGS. 17C and 17D, a range 1711 is photographed, and a longitudinally contracted photographed image 1712 is stored. Accordingly, as illustrated in FIGS. 17E and 17F, an image 1714 which is contraction-corrected by clipping out a range 1713 while shifting the read-out position of each of the lines by using the detection result of the motion amount, is output. Moreover, if the camera is tilting upwards, as illustrated in FIGS. 17G and 17H, a range 1721 is photographed, and a longitudinally expanded photographed image 1722 is stored. Accordingly, as illustrated in FIGS. 17I and 17J, an image 1724 which is expansion-corrected by clipping out a range 1723 while shifting the read-out position of each of the lines by using a detection result of the motion amount, is output.
Japanese Patent Application Laid-Open No. 2006-186885 describes a camera arranged to perform correction by bringing a clipping-out position of an output image to the center in order to avoid the situation in which the clipping-out position is very close to the edge of a photographed image when camera shaking becomes large so that remain no margin of the angle of view remains and distortion cannot be corrected.
Moreover, Japanese Patent Application Laid-Open No. 2007-020045 describes a camera arranged to correct distortion including only expansion and contraction in a buffer memory by avoiding distortion in any diagonal direction in a photographed image by replacing the horizontal scanning and the vertical scanning of an image pickup sensor with each other according to the panning or the tilting of a camera.
However, because the conventional correction methods each clip out a part of an image to output the part, the angle of view of the output image becomes narrower than that of a photographed image.
For example, some digital cameras of a lens-exchangeable type make their abilities to record the angles of views capable of being photographed with their lenses a major advantage. Moreover, some digital cameras of a lens-integrated type are configured to have short focal lengths on their wide angle sides and also make their abilities to photograph objects at wide angles a major advantage. However, if a part of an image is clipped out, these features of the cameras become useless. Furthermore, if a part of an image is clipped out, the number of effective pixels of an image pickup sensor decreases, and consequently it becomes difficult to photograph a higher definition image in comparison with the case of performing no correction.
Moreover, the replacement of the scanning directions in an image pickup by the rolling-shutter method makes the read-out and the processing mechanism of an image pickup signal from an image pickup sensor complex.